Method of teaching robot and robot system

ABSTRACT

A robot system includes a robot, a vision sensor, and a controller. The vision sensor is configured to be detachably attached to the robot. The controller is configured to measure a reference object by using the vision sensor and calibrate a relative relationship between a sensor portion of the vision sensor and an engagement portion of the vision sensor, and teach the robot by referring to the relative relationship and by using the vision sensor, after the vision sensor is attached to the robot.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to teaching a robot.

Description of the Related Art

In factories, a robot system is used to automate work, such as transferof parts, assembly, and processing. The robot system has a robot whichhas multiple degrees of freedom. Examples of the robot having multipledegrees of freedom include a vertically articulated robot, ahorizontally articulated robot, a Cartesian coordinate robot, and aparallel link robot. A user can cause a robot to move in accordance witha robot program, and thereby can cause the robot to perform a variety oftypes of work for respective purposes.

In order to cause a robot to perform production work, such as transferof workpieces and assembly, a user needs to teach a robot a posture ofthe robot in advance. Teaching a robot is typically performed by a userusing a teaching pendant to perform jog feed of the robot to cause therobot to take a posture for work and causing a storage unit to store theposture. When a user causes a robot to perform work which needsprecision, the user needs to determine a relative position andorientation, with high accuracy, between an object such as a workpieceor a workpiece holding jig, and an end effector such as a hand or a toolattached to the robot, and teach the robot the relative position andorientation.

Japanese Patent Application Publication No. H7-84631 proposes atechnique to correct a teach position of a robot. In this technique, athree-dimensional vision sensor is attached to a welding robot in placeof, or in parallel with a welding torch, a mark formed on an object tobe worked is measured by using the vision sensor, and thereby a teachposition of the robot is corrected. The technique of Japanese PatentApplication Publication No. H7-84631 calibrates the relationship betweenthe robot and the vision sensor, and corrects the teach position byusing information on a mark position seen from the vision sensor, in astate where the vision sensor is attached to the robot.

However, because the robot has an error in its posture, the visionsensor attached to the robot also has an error in its position andorientation. As a result, in the technique of Japanese PatentApplication Publication H7-84631, the error in posture of the robotaffects the accuracy of the calibration, thus reducing the accuracy ofteaching the robot.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, a method ofteaching a robot includes measuring a reference object by using a visionsensor and calibrating a relative relationship between a sensor portionof the vision sensor and an engagement portion of the vision sensor,attaching the engagement portion of the vision sensor to a robot, afterthe calibrating, and teaching the robot by referring to the relativerelationship and by using the vision sensor.

According to a second aspect of the present invention, a robot systemincludes a robot, a vision sensor configured to be detachably attachedto the robot, and a controller. The controller is configured to measurea reference object by using the vision sensor and calibrate a relativerelationship between a sensor portion of the vision sensor and anengagement portion of the vision sensor, and teach the robot byreferring to the relative relationship and by using the vision sensor,after the vision sensor is attached to the robot.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory diagram of a robot system of a firstembodiment.

FIG. 2A is a perspective view of a workpiece holding jig of the firstembodiment.

FIG. 2B is a perspective view of the workpiece holding jig, a firstworkpiece, and a second workpiece of the first embodiment.

FIG. 3A is a perspective view of a stereo camera of the firstembodiment.

FIG. 3B is a perspective view of the stereo camera of the firstembodiment.

FIG. 3C is a cross-sectional view of the stereo camera of the firstembodiment.

FIG. 4A is a perspective view of a robot hand and the stereo camera ofthe first embodiment.

FIG. 4B is a perspective view of the robot hand and the stereo camera ofthe first embodiment.

FIG. 4C is a perspective view of the robot hand and the stereo camera ofthe first embodiment.

FIG. 5 is a flowchart illustrating a method of teaching a robot,according to the first embodiment.

FIG. 6 is a flowchart illustrating a method of teaching the robot,according to the first embodiment.

FIG. 7A is an explanatory diagram of a calibration jig of the firstembodiment.

FIG. 7B is a plan view of a reference pattern portion of the firstembodiment.

FIG. 8 is a schematic diagram for illustrating a relationship betweentarget values and measured values of feature points.

FIG. 9A is an explanatory diagram illustrating a relationship between atool coordinate system positioned at a leading end of a robot arm and abase coordinate system of the stereo camera.

FIG. 9B is an explanatory diagram illustrating a relationship between atool coordinate system positioned at the leading end of the robot armand the base coordinate system of the stereo camera.

FIG. 10 is an explanatory diagram of a robot system of a secondembodiment.

FIG. 11A is a front view illustrating a state where a stereo camera isheld by a robot hand in the second embodiment.

FIG. 11B is a side view illustrating the state where the stereo camerais held by the robot hand in the second embodiment.

FIG. 12 is a flowchart illustrating a method of teaching a robot,according to the second embodiment.

FIG. 13A is an explanatory diagram of a calibration jig of a thirdembodiment.

FIG. 13B is an explanatory diagram of a robot of the third embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, some embodiments of the present invention will be describedin detail with reference to the accompanying drawings.

First Embodiment

FIG. 1 is an explanatory diagram of a robot system of a firstembodiment. A robot system 100 includes a robot 200, a stereo camera 300which is one example of vision sensors, a workpiece holding jig 400, arobot control device 500, a sensor control device 600, and a display700. The robot system 100 also includes a teaching pendant 800 and aninput device 850, which are one example of operation devices. The robot200 includes a robot arm 201, and a robot hand 202 which is one exampleof end effectors. The robot arm 201 is a vertically articulated robotarm. The base end of the robot 200, that is, the base end of the robotarm 201 is fixed to a stand 150.

The leading end of the robot arm 201 is provided with the robot hand 202attached thereto, directly or via a force sensor (not illustrated). Thatis, the leading end of the robot 200, which serves as a hand of therobot 200, is the robot hand 202. The robot hand 202 includes a palmunit 210 which is a hand body, and a plurality of fingers 211 and 212which can open and close with respect to the palm unit 210. In the firstembodiment, the number of the fingers is two. With these components, therobot hand 202 can hold or release a workpiece. The palm unit 210includes a housing and a driving mechanism which is housed by thehousing and configured to drive the fingers 211 and 212.

The workpiece holding jig 400 is a jig which chucks a workpiece so thatthe workpiece does not move with respect to the stand 150, and is fixedto the stand 150. The workpiece holding jig 400 is disposed in an areawithin which the robot hand 202 can move. The leading end of the robot200, that is, the robot hand 202 can move with six degrees of freedom,in a robot coordinate system O whose origin is at the base end of therobot 200. Specifically, the robot hand 202 can move in threetranslational directions and three rotational directions, with themotion of the robot arm 201. The translational directions correspond tothree axes which are orthogonal to each other in the robot coordinatesystem O, and the rotational directions are directions around the threeaxes.

The teaching pendant 800 is used to send a command to the robot controldevice 500 through an operation by a user. Upon receiving the command,the robot control device 500 causes the robot 200 to move, depending onthe command. That is, through the operation of the teaching pendant 800,the user can move the robot arm 201, and perform a jog feed of the robothand 202 in any direction and at any speed. Also, through the operationof the teaching pendant 800, the user allows the robot 200 to open andclose the fingers 211 and 212 of the robot hand 202.

The robot control device 500 mainly controls the motion of the robot200. The sensor control device 600 mainly controls the operation of thestereo camera 300, and performs computing processes, such as imageprocessing and a measurement process.

The robot control device 500 is a computer, and includes a centralprocessing unit (CPU) 501. In addition, the robot control device 500includes, as example of memories, a read only memory (ROM) 502, a randomaccess memory (RAM) 503, and a hard disk drive (HDD) 504. The robotcontrol device 500 also includes an interface (I/F) 505 and a bus 506.The CPU 501, the ROM 502, the RAM 503, the HDD 504, and the interface505 are communicatively connected with each other via the bus 506.

The sensor control device 600 is a computer, and includes a CPU 601. Inaddition, the sensor control device 600 includes, as example ofmemories, a ROM 602, a RAM 603, and an HDD 604. The sensor controldevice 600 also includes a recording-disk drive 605, an interface (I/F)606, and a bus 607. The CPU 601, the ROM 602, the RAM 603, the HDD 604,the disk drive 605, and the interface 606 are communicatively connectedwith each other via the bus 607.

The interface 505 of the robot control device 500 is connected to therobot 200, the workpiece holding jig 400, and the teaching pendant 800via communication lines. The interface 606 of the sensor control device600 is electrically connected to the stereo camera 300, the display 700,and the input device 850 via communication lines. The first embodimenthas the configuration in which a user operates the teaching pendant 800while watching a display screen of the display 700. Thus, the robotcontrol device 500 and the sensor control device 600 may not beelectrically connected with each other via a communication line or thelike.

The ROM 502 of the robot control device 500 stores a base program. TheRAM 503 is a memory to temporarily store various data, such as resultsof a computing process performed by the CPU 501. The HDD 504 stores arobot program 510 which defines the motion of the robot 200. The robotprogram 510 includes teach point information 511 and command information512. The CPU 501 controls the motion of the robot 200 depending on therobot program 510, and causes the robot 200 to perform work, such asworkpiece assembly work. When setting the teach point information 511,that is, when performing teaching work, the CPU 501 controls the motionof the robot 200 depending on a command from the teaching pendant 800.In addition, the CPU 501 can create or update the posture information,which is used when the robot 200 is actually moved, as the teach pointinformation 511.

The ROM 602 of the sensor control device 600 stores a base program. TheRAM 603 is a memory to temporarily store various data, such as resultsof a computing process performed by the CPU 601. The HDD 604 stores aprogram 610 used when the teaching is performed. The disk drive 605 canread various data or a program recorded on a disk 611. When teaching therobot 200, the CPU 601 controls the operation of the stereo camera 300and controls the motion of the robot 200 via the robot control device500, depending on the program 610. In addition, the CPU 601 controlsdisplaying operation of the display 700. For example, the CPU 601 causesthe display 700 to display image data captured by the stereo camera 300and resultant data obtained by executing image processing, and providesinformation to a user. The input device 850 may be a keyboard or amouse, and can output information, such as characters, numerical values,or a pointer position, to the sensor control device 600 by a useroperating the input device 850 while watching the display 700.

The interface 505 of the robot control device 500 and the interface 606of the sensor control device 600 can be electrically connected to anexternal memory (not illustrated). In the first embodiment, thedescription will be made for a case where the HDD 604 is acomputer-readable recording medium and stores the program 610. However,the present disclosure is not limited to this. The program 610 may berecorded in any recording medium as long as the recording medium is acomputer-readable recording medium. For example, the ROM 602, the disk611, or an external memory (not illustrated) may be used as therecording medium that provides the program 610. Specifically, a flexibledisk, an optical disk such as a CD-ROM or a DVD-ROM, a magneto-opticaldisk, a magnetic tape, a nonvolatile memory such as a USB memory, anSSD, or the like may be used as the recording medium.

Also in the first embodiment, the description has been made for the casewhere the robot control device 500 and the sensor control device 600 areeach achieved by a single computer, and thus the robot system 100 hastwo computers in total. However, the present disclosure is not limitedto this. Both functions of the robot control device 500 and the sensorcontrol device 600 may be achieved by a single computer, or may beachieved by three or more computers which perform distributedprocessing. Furthermore, although the sensor control device 600 isdisposed outside the housing of the stereo camera 300, a smart camerahoused by the housing may achieve the same function.

FIG. 2A is a perspective view of a workpiece holding jig 400. FIG. 2B isa perspective view of the workpiece holding jig 400, a workpiece W1which is a first workpiece, and a workpiece W2 which is a secondworkpiece. As illustrated in FIG. 2A, the workpiece holding jig 400includes a base portion 401, work butted-against portions 402 and 403which are fixed to the base portion 401, and jig fingers 404 and 405.The workpiece holding jig 400 can open and close the jig fingers 404 and405, depending on the robot program 510. The workpiece W1 can be chuckedby closing the jig fingers 404 and 405 in a state where the workpiece W1is placed on the base portion 401.

FIG. 2B illustrates a state where the workpiece W1 is held by theworkpiece holding jig 400. In the first embodiment, fitting work for theworkpiece W1 and the workpiece W2 is performed as assembly work, inwhich the workpiece W2 is assembled to the workpiece W1. Specifically,the fitting work is performed by causing the workpiece holding jig 400to hold the workpiece W1, causing the robot hand 202 to hold theworkpiece W2, and moving the robot arm 201. As illustrated in FIG. 2B,when the fitting work is performed, the robot hand 202 holding theworkpiece W2 is moved to a work start position PA which is directlyabove the workpiece W1, and then the workpiece W2 is moved from the workstart position PA to a position directly below the work start positionPA. Thus, a correct teaching is required to position the robot hand 202correctly at the work start position PA, that is, to position theworkpiece W2 correctly.

In the first embodiment, the workpiece holding jig 400 is a measuredobject used when the teaching work is performed. The workpiece holdingjig 400 illustrated in FIG. 2A is provided with a plurality of marks MK1to MK3. The marks MK1 to MK3 can be measured by the stereo camera 300,and are one example of feature points. The marks MK1 to MK3 arepositioned with respect to mechanical references of the workpieceholding jig 400, with high accuracy. The mechanical references may bethe work butted-against portions 402 and 403. The marks MK1 to MK3 areformed on the top face of the base portion 401, and have a predeterminedprecision range of thickness or depth in a height direction. The marksMK1 to MK3 are black so that they have a sufficient contrast to the baseportion 401, and thus can be recognized with high accuracy as thefeature points, through image processing. Here, any method may be usedto provide the marks MK1 to MK3 to the base portion 401. For example,laser beam machining, printing, etching, plating, or sticking seals maybe used to provide the marks. In addition, although the example is givenfor the case where the marks MK1 to MK3 are directly provided to theworkpiece holding jig 400, the marks MK1 to MK3 may be temporarilyarranged only in the teaching, by providing the marks to a referenceplate which is another member, and attaching the reference plate to theworkpiece holding jig 400.

In the first embodiment, the stereo camera 300 is used to teach therobot 200. The stereo camera 300 is detachably attached to the robot200. Specifically, the stereo camera 300 is held by the robot hand 202in a state where the stereo camera 300 is positioned with respect to therobot hand 202.

FIGS. 3A and 3B are perspective views of the stereo camera 300. FIG. 3Cis a cross-sectional view of the stereo camera 300. The stereo camera300 is a camera which captures images of a measured object by using thestereo method so that a three-dimensional position and orientation ofthe measured object can be measured. The stereo camera 300 can captureimages upon receiving an imaging command from the sensor control device600, and send the acquired image data to the sensor control device 600.The stereo camera 300 includes a sensor portion 305. The sensor portion305 includes a camera 301 which is a first camera, and a camera 302which is a second camera. The two cameras, 301 and 302, are disposed inthe interior of a housing 310. The cameras 301 and 302 are digitalcameras having an imaging device, such as a CCD image sensor or a CMOSimage sensor. The housing 310 is fixed to a plate-like base portion 311via an attachment portion 340. The base portion 311 is a member which isto be attached to the robot hand 202. The base portion 311 is providedwith positioning pins 321 and 322, and tapered portions 331 and 332, asan attaching and detaching mechanism to/from the robot hand 202. Thepositioning pins 321 and 322, and the tapered portions 331 and 332 serveas engagement portions.

As illustrated in FIG. 3C, the cameras 301 and 302 are disposedsymmetrically to each other, each tilted by a convergence angle α withrespect to a symmetry plane A-A of these optical systems. Providing theconvergence angle a allows the stereo camera 300 to have a wide commonfield of view, together with a relatively short working distance. Thebottom face of the base portion 311 is orthogonal to the symmetry planeA-A. In addition, the positioning pins 321 and 322, and the taperedportions 331 and 332 are also disposed so as to be symmetrical withrespect to the symmetry plane A-A.

Camera parameters for the stereo camera 300 have been measured inadvance through a stereo camera calibration. Specifically, camerainternal parameters have been measured for the cameras 301 and 302. Oneof the camera internal parameters indicates a relationship between apixel coordinate value, expressed in units of pixels, of an imagecaptured by the camera 301 and a line-of-sight vector in athree-dimensional space; the other of the camera internal parametersindicates a relationship between a pixel coordinate value, expressed inunits of pixels, of an image captured by the camera 302 and theline-of-sight vector. In addition, a stereo camera external parameterhas also been measured. The stereo camera external parameter indicates arelative position and orientation between the camera 301 and the camera302. That is, there has been calculated a coordinate transformationmatrix which indicates a relative position and orientation between asensor coordinate system V1, which represents a position and orientationof the camera 301, and a sensor coordinate system V2, which represents aposition and orientation of the camera 302. The sensor coordinate systemV1 has the origin which is the lens principal point of the camera 301,and the sensor coordinate system V2 has the origin which is the lensprincipal point of the camera 302. The stereo camera calibration can beperformed by using any of various known methods.

The CPU 601 extracts feature points from image data captured by thecamera 301 and image data captured by the camera 302, and thereby candetermine three-dimensional coordinate values of the feature points withrespect to the sensor coordinate system. In the first embodiment, thecamera 301 is a reference camera. Thus, in the first embodiment, thesensor coordinate system V1, which is of the sensor coordinate systemsV1 and V2 of the sensor portion 305, is a sensor coordinate system Vwhich is a first coordinate system, and which is used for the sensorportion 305, as a representative. Thus, results of the stereomeasurement are expressed by coordinate values obtained with respect tothe sensor coordinate system V of the camera 301. The position andorientation of feature points with respect to the sensor coordinatesystem V can be three-dimensionally measured with high accuracy by usinga known stereo camera calibration method, but cannot be measureddirectly from the outside. Thus, in the first embodiment, calibrationbetween base and camera is also performed to determine the position andorientation of the sensor coordinate system V with respect to the baseportion 311. The calibration data obtained from the calibration betweenbase and camera is stored in the HDD 604, which is a memory used for thesensor control device 600.

FIGS. 4A, 4B, and 4C are perspective views of the robot hand 202 and thestereo camera 300 of the first embodiment. FIGS. 4A and 4B illustrate astate where the stereo camera 300 is released by the robot hand 202.FIG. 4C illustrates a state where the stereo camera 300 is held by therobot hand 202.

A flat surface 210A of a palm unit 210 of the robot hand 202 is providedwith a round hole 221 and an elongated hole 222 at positionscorresponding to the positioning pins 321 and 322 of the stereo camera300. The round hole 221 and the elongated hole 222 serve as anengagement portion. The flat surface 210A of the palm unit 210 of therobot hand 202, the round hole 221, and the elongated hole 222 aremechanical references of the robot hand 202, and the fingers 211 and 212are mounted within a predetermined tolerance with respect to themechanical references. Specifically, in a state where a fingeradjustment jig (not illustrated) is positioned with respect to the palmunit 210 by using the flat surface 210A of the palm unit 210, the roundhole 221, and the elongated hole 222, the installation positions of thefingers 211 and 212 in an opening and closing direction of the fingers211 and 212 are adjusted with respect to the finger adjustment jig (notillustrated). The dimension between the fingers 211 and 212 in theopening and closing direction can be adjusted by attaching shims, havingdifferent thicknesses, to an opening and closing mechanism (notillustrated) so as to sandwich the fingers. In this manner, the fingers211 and 212 are adjusted with respect to the palm unit 210. When thebase portion 311 for the stereo camera 300 is pressed against the palmunit 210 of the robot hand 202, the surface 210A of the palm unit 210and the surface 311A of the base portion 311 abut against each other,and the positioning pins 321 and 322 fit with the round hole 221 and theelongated hole 222, respectively. With the engagement between thepositioning pins 321 and 322, which serves as the engagement portion ofthe stereo camera 300, and the holes 221 and 222, which serve as theengagement portion of the robot hand 202, the stereo camera 300 can bepositioned with respect to the robot hand 202 with high accuracy. Insidethe fingers 211 and 212, tapered portions 231 and 232 are disposed. Whenthe fingers 211 and 212 are closed in a state where the base portion 311is pressed against the palm unit 210, the tapered portions 231 and 232of the robot hand 202 engage with the tapered portions 331 and 332 ofthe stereo camera 300, respectively. The tapered portions 231 and 232,and 331 and 332 produce a force which presses the base portion 311against the palm unit 210, so that the stereo camera 300 can be stablyfixed to the robot hand 202. Thus, the attaching and detaching mechanismis configured to directly attach the stereo camera 300 to the robot hand202.

With the above-described configuration, the stereo camera 300 can beattached to or detached from the robot hand 202 by only closing oropening the fingers 211 and 212 of the robot hand 202. In addition, thestereo camera 300 can be positioned with respect to the robot hand 202with high accuracy, by only causing the robot hand 202 to hold thestereo camera 300. In addition, the stereo camera 300 is directlyattached to the round hole 221 and the elongated hole 222, which aremechanical references of the robot hand 202. Thus, when the teachingwork is performed, this configuration reduces influence caused by anerror of the robot arm 201, and an error produced in production of aportion of the robot hand 202 between a palm unit 210 and a flangesurface. The flange surface is the leading end of the robot arm 201. Asa result, the teaching work can be performed with high accuracy.

The stereo camera 300 is pressed against the palm unit 210 of the robothand 202 via the tapered portion 331 and 332. With this configuration,the teaching can be steadily performed without the robot hand 202loosening its grip even when the posture of the robot hand 202 changes.In addition, the symmetry plane A-A of the optical systems of the stereocamera 300 coincides with a symmetry plane of the positioning pins 321and 322, and with a symmetry plane of the tapered portions 331 and 332.With such an arrangement, even when the base portion 311 is distortedbecause the tapered portions 331 and 332 are pushed toward the palm unit210 by the tapered portions 231 and 232, the influence by the distortionis caused symmetrically. This prevents the error from easily occurringduring the teaching. In addition, the housing 310 and the base portion311 are formed of members different from each other, and joined witheach other via the attachment portion 340. With such a configuration,even when the base portion 311 is distorted as described above, therelative position and orientation between the camera 301 and the camera302 of the stereo camera 300 hardly changes. In general, a slight changein relative position and orientation between a pair of cameras of astereo camera causes a measured value, obtained by the stereo camera, tohave a large error. However, the configuration of the stereo camera 300of the first embodiment produces less error than a configuration inwhich the robot hand 202 directly holds the housing 310. Thus, the errorin the teaching can be reduced even though the stereo camera 300 isdesigned so as to be light in weight, that is, to have a low stiffness.In addition, an air gap is created, as a heat insulation layer, betweenthe housing 310 and the base portion 311. This can prevent heat,produced in the robot hand 202, from being easily transmitted to thehousing 310, thus reducing the error caused by thermal deformation ofthe stereo camera 300. Furthermore, in a case where the design of therobot hand 202 is changed, or where the stereo camera 300 is to beattached to another robot, the identical stereo camera 300 can be usedby changing the design of the base portion 311.

FIGS. 5 and 6 are flowcharts illustrating a method of teaching the robot200, according to the first embodiment. Here, the flowcharts of FIGS. 5and 6 include user operations in addition to the processing performed bythe CPUs 501 and 601. In FIG. 5, the process first calibrates therelative relationship between a base coordinate system B and the sensorcoordinate system V (S101: calibration step). The base coordinate systemB is a second coordinate system, and has its reference which is at theengagement portion of the stereo camera 300, used to attach the stereocamera 300 to the robot 200. In the first embodiment, the origin of thebase coordinate system B is at one of the positioning pins 321 and 322of the base portion 311 of the stereo camera 300. The positioning pins321 and 322 are the engagement portion. As an example, the origin is setat a base portion of the positioning pin 321.

The calibration of the relative relationship between the sensor portion305 of the stereo camera 300 and the positioning pin 321, that is, thecalibration of the relative relationship between the sensor coordinatesystem V and the base coordinate system B is performed in a state wherethe stereo camera 300 is removed from the robot 200, and is set on acalibration jig. The position and orientation of feature points withrespect to the sensor coordinate system V can be three-dimensionallymeasured with high accuracy. However, the sensor coordinate system V isa coordinate system whose origin is at the lens principal point of thecamera 301. In the first embodiment, the CPU 601 performs thecalibration to identify the relative position and orientation betweenthe base coordinate system B and the sensor coordinate system V, andcauses the HDD 604 to store the calibration data. Hereinafter, themethod of the calibration will be specifically described.

FIG. 7A is an explanatory diagram of a calibration jig 900 used for thecalibration. The calibration jig 900 includes a base portion 901, pillarportions 902 and 903, a top plate portion 904, and a reference patternportion 905 which is a reference object. The base portion 901 and thetop plate portion 904 are plate-like members which are disposed facingeach other, and joined with each other via the pillar portions 902 and903. The base portion 901 is provided with holes 906 and 907 atpositions corresponding to the pins 321 and 322 of the stereo camera300, as is the round hole 221 and the elongated hole 222 of the palmunit 210. With this structure, the stereo camera 300 can be positionedwith respect to the reference pattern portion 905 with highreproducibility. When the stereo camera 300 is set on the calibrationjig 900, the field of view of the stereo camera 300 contains thereference pattern portion 905.

FIG. 7B is a plan view of the reference pattern portion 905. Thereference pattern portion 905 is disposed on the inner surface of thetop plate portion 904. The reference pattern portion 905 is a plate-likemember. On the surface of the reference pattern portion 905, N number ofmarks 908, which are made with high accuracy through an etching processor the like, are formed like an array. Positions of the top face of thebase portion 901, the holes 906 and 907, and the marks 908 of thereference pattern portion 905 are measured with a measuring instrument(not illustrated), with high accuracy, after the calibration jig 900 ismade. Furthermore, the base coordinate system B is set with respect tothe measured values of the top face of the base portion 901 and theholes 906 and 907. Three-dimensional coordinate values m[i] (i=1 to N)of the marks 908 have been coordinate-transformed to three-dimensionalcoordinate values ^(B)m[i] (i=1 to N) with respect to the basecoordinate system B, and the coordinate-transformed three-dimensionalcoordinate values ^(B)m[i] (i=1 to N) are stored in the HDD 604. The CPU601 calibrates the relative relationship between the base coordinatesystem B and the sensor coordinate system V by measuring the marks 908of the reference pattern portion 905 in stereo, in a state where thestereo camera 300 is positioned with respect to the calibration jig 900.

In the first embodiment, the CPU 601 determines a rotation matrix R_(c),which is a coordinate transformation matrix, and a translation vectort_(c), as the relative relationship between the base coordinate system Band the sensor coordinate system V. If coordinate values ^(V)m[i] (i=1to N) of the marks 908 with respect to the sensor coordinate system V,obtained when the marks 908 are measured in stereo, have no errors, thefollowing expression (1) holds.

^(B) m[i]=R _(c)·^(V) m[i]+t _(c) (i=1, 2, 3, . . . , N)   (1)

However, since actual measurement data has errors, the left side and theright side of the expression (1) are not exactly equal to each other.Thus, the CPU 601 determines the rotation matrix R_(c) and thetranslation vector t_(c) which minimize the following expression (2).That is, the CPU 601 determines least squares solutions of the followingexpression (2).

$\begin{matrix}{\sum\limits_{i = 1}^{N}{{{{\,^{B}m}\lbrack i\rbrack} - \left( {{R_{C} \cdot {{\,^{v}m}\lbrack i\rbrack}} + t_{C}} \right)}}^{2}} & (2)\end{matrix}$

This problem is known as the matching problem between point sets, andcan be solved by using, for example, an approach which uses singularvalue decomposition. By using the above method, the rotation matrixR_(c) and the translation vector t_(c), which are the relativerelationship between the sensor coordinate system V and the basecoordinate system B, can be calibrated with high accuracy. The HDD 604of the sensor control device 600 stores data of the rotation matrixR_(c) and the translation vector t_(c), calibrated in this manner.

A user then sets target coordinate values for the teaching (S102). Inthe first embodiment, the user sets the target coordinate values atwhich the workpiece W2 is positioned at the position PA, as illustratedin FIG. 2B. The setting of the target values is performed by inputting apositional relationship in design between the mechanical references ofthe stereo camera 300, such as the surface of the base portion 311 andthe positioning pins 321 and 322, and the marks MK1 to MK3 provided tothe workpiece holding jig 400. That is, the user sets target values^(B)p[i] (i=1 to 3) of three-dimensional coordinates of the marks MK1 toMK3, in design, with respect to the base coordinate system B. Here, thetarget values ^(B)p[i] (i=1 to 3) are three-dimensional coordinatevalues, each expressed by a vector having three components of XYZ. Thetarget values ^(B)p[i] may be set by the user selecting the targetvalues ^(B)p[i] from a list of target values, which is stored in advancein the HDD 604 of the sensor control device 600, or may be set by theuser operating the input device 850 and inputting numerical values intothe input device 850. In the case where a user uses the input device 850to input numerical value, a marker coordinate system M representing aposition and orientation of the marks MK1 to MK3 may be introduced tomore simplify the input operation. In this case, the user inputsthree-dimensional coordinate values ^(M)p[i] (i=1 to 3) of the marks MK1to MK3 with respect to the marker coordinate system M, and design valuesof a relative position and orientation of the marker coordinate system Mwith respect to the base coordinate system B. Where the design valuesare expressed by a rotation component of ^(B)R_(M) and a translationcomponent of ^(B)t_(M), the three-dimensional coordinate values ^(B)p[i](i=1 to 3) of the marks MK1 to MK3 are expressed by the followingexpression (3).

^(B) p[i]=^(B) R _(M)·^(M) p[i]+^(B) t _(M) (i=1, 2, 3)   (3)

With the input operation performed in the marker coordinate system M,the arrangement determined by only a shape of the marks MK1 to MK3 and aposition and orientation of the robot hand 202 with respect to themarker coordinate system M can be set separately. This simplifies thesetting when the number of the marks is large. Thus, Step S102 causesthe HDD 604 of the sensor control device 600 to store the target values^(B)p[i] of the marks MK1 to MK3 with respect to the base coordinatesystem B.

The user then sets a threshold δ to the sensor control device 600(S103). The threshold δ is used for a later-described convergencedetermination. The user can freely set the thresholds δ in considerationof accuracy required for an intended use. Thus, Step S103 causes the HDD604 of the sensor control device 600 to store the threshold δ.

Then the stereo camera 300 is attached to the robot 200 (S104:attachment step). Specifically, the user causes the robot hand 202 tohold the stereo camera 300. For example, the user performs jog feed ofthe robot arm 201 by operating the teaching pendant 800 so that the palmunit 210 of the robot hand 202 faces upward, and then places the stereocamera 300 on the flat surface 210A, which is a palm of the palm unit210. The user then operates the teaching pendant 800, and causes thefingers 211 and 212 of the robot hand 202 to close to chuck the stereocamera 300. The stereo camera 300 is held by the robot hand 202, andthereby positioned with respect to the robot hand 202 with highaccuracy. Thus, the base coordinate system B is positioned with respectto the robot hand 202 with high accuracy.

Then, the CPU 501 and the CPU 601 work with each other to refer to therelative relationship R_(c) and t_(c), which is calibrated and stored inthe HDD 604, and to teach the robot 200 by using the stereo camera 300(S105 to S109, and S201 to S208). Hereinafter, this teaching step willbe specifically described. The CPU 501 moves the robot 200, that is, therobot arm 201 so as to take a posture in which the teaching is started(S105). That is, the CPU 501 moves the stereo camera 300 to a positionat which the teaching is started. The posture of the robot 200 in whichthe teaching is started is a posture in which the field of view of thestereo camera 300 contains the workpiece holding jig 400, which is ameasured object. That is, the robot 200, that is, the robot arm 201 maytake any posture as long as the stereo camera 300 can capture images ofthe marks MK1 to MK3 of the workpiece holding jig 400. In this case, theCPU 501 may move the robot 200 depending on a command obtained when theuser operates the teaching pendant 800, or may move the robot 200 so asto take a posture which has been set in advance by an off-line teachingor the like. After moving the robot 200 to the teach point, if the marksMK1 to MK3 are positioned outside the field of view of the stereo camera300, the user can perform jog feed of the robot arm 201 while checkingthe image from the stereo camera 300 on the display 700. Step 105 allowsthe field of view of the stereo camera 300 to contain the three marksMK1 to MK3.

If the user performs an on-operation by using the input device (S106),the CPU 601 of the sensor control device 600 repeats processesillustrated in FIG. 6, until the user performs an off-operation by usingthe input device 850. The on-operation and the off-operation may beperformed by the user moving a cursor to a corresponding key of asoftware keyboard displayed on the display 700 and clicking the key byusing a mouse or the like, or pressing a corresponding button of theinput device 850.

Hereinafter, repetitive processes performed when the on-operation isperformed will be described. The CPU 601 sends an imaging command to thestereo camera 300, causes the stereo camera 300 to capture images of theworkpiece holding jig 400 in which the marks MK1 to MK3 are contained,and obtains the images from the stereo camera 300 (S201). Thus, the CPU601 obtains the images (stereo images) from both the camera 301 and thecamera 302. That is, the CPU 601 obtains visual information from thestereo camera 300.

The CPU 601 then performs image processing and a stereo calculationprocess on the obtained stereo images, and measures positions of themarks MK1 to MK3 (S202). The image processing for measuring thepositions of the marks MK1 to MK3 can use a variety of known methods.For example, the image processing performs an edge extraction process oneach of the stereo images, and uses shape feature values, such ascircularity or circumradius of the extracted edge, to select only theedges of the marks MK1 to MK3. Then, the image processing performs anellipse fitting process on the edges of the marks MK1 to MK3 of theimages, and determines pixel coordinates of the center points of themarks MK1 to MK3. Here, not the circle fitting process but the ellipsefitting process is used, because the circular marks MK1 to MK3 can bedeformed and have an ellipse-like shape when images of the marks arecaptured due to the relative arrangement of the stereo camera 300 andthe marks MK1 to MK3. After determining the pixel coordinates of thecenter points of the circles of the marks MK1 to MK3, the imageprocessing associates the pixel coordinates of the image captured by thecamera 301, with the pixel coordinates of the image captured by thecamera 302, and calculates three-dimensional coordinate values. Theimage processing can calculate the coordinate values of the marks MK1 toMK3 with respect to the sensor coordinate system V, by using the camerainternal parameters and the camera external parameter of the cameras 301and 302. The coordinate values are expressed as ^(V)q[i] (i=1 to 3).Thus, the CPU 601 extracts feature points contained in the visualinformation, which is obtained from the stereo camera 300.

The CPU 601 then refers to the rotation matrix R_(c) and the translationvector t_(c) stored in the HDD 604, and coordinate-transforms thecoordinate values ^(V)q[i] (i=1 to 3) of the marks MK1 to MK3 tomeasured values ^(B)q[i] (i=1 to 3) with respect to the base coordinatesystem B (S203). That is, the CPU 601 determines the measured values^(B)q[i] (i=1 to 3) by using the following expression (4), as a resultof the measurement using the stereo camera 300.

^(B) q[i]=R _(c)·^(V) q[i]+t _(c) (i=1,2,3)   (4)

The CPU 601 then uses the measured values ^(B)q[i] (i=1 to 3) withrespect to the base coordinate system B, to which the coordinate valueshas been coordinate-transformed, and target values ^(B)p[i] (i=1 to 3),and computes the amount of positional difference between the measuredvalues and the target values (S204). The comparison between thecoordinate values ^(B)p[i] and ^(B)q[i] is easy, because the coordinatevalues ^(B)p[i] and ^(B)q[i] are coordinate values with respect to thebase coordinate system B, obtained through the coordinatetransformation. The CPU 601 uses an expression (5), and computesdifference vectors Δp[i] (i=1 to 3) between the target values ^(B)p[i]and the measured values ^(B)q[i].

Δp[i]=^(B) q[i]−^(B) p[i] (i=1,2,3)   (5)

In addition, the CPU 601 also computes the amount of shift of the basecoordinate system B, as another index for the amount of positionaldifference. Where a coordinate system B′ is a coordinate system obtainedafter the base coordinate system B moves by a rotation matrix R and atranslation vector t, target values ^(B)p′[i] obtained after the basecoordinate system B moves are expressed as the following expression (6).

^(B) p′[i]=R· ^(B) p[i]+t (i=1,2,3)   (6)

Thus, in order to shift the base coordinate system B so that the targetvalues ^(B)p′[i] obtained after the base coordinate system B is shiftedare equal to the measured values ^(B)q[i] of the marks MK1 to MK3, leastsquares solutions of the rotation matrix R and the translation vector tare determined so that the following expression (7) has the minimumvalue. That is, because the measured values ^(B)q[i] of the marks MK1 toMK3 have measurement errors which depend on the stereo camera 300 andthe individual difference of the marks MK1 to MK3 and which differs fromeach other, the least squares solutions of the rotation matrix R and thetranslation vector t are determined. Here, N is the number of the markswhich are the feature points, and i is an integer from 1 to N. That is,the CPU 601 determines the rotation matrix R and the translation vectort so that the expression (7) has the minimum value.

$\begin{matrix}{\sum\limits_{i = 1}^{N}{{{{\,^{B}q}\lbrack i\rbrack} - \left( {{R \cdot {{\,{\,^{B}p}}\lbrack i\rbrack}} + t} \right)}}^{2}} & (7)\end{matrix}$

This computation can be performed by using the same method as that usedfor the calibration between base and camera, and that used for thematching problem between point sets. With the above-described computingprocess, there is determined the rotation matrix R and the translationvector t which indicate the positional difference of the base coordinatesystem B. FIG. 8 is a schematic diagram for illustrating a relationshipbetween the target values ^(B)p[i] of the feature points and themeasured values ^(B)q[i] of the feature points. In FIG. 8, the targetvalues ^(B)p[i] (i=1 to 3) are expressed as p1, p2, and p3, and themeasured values ^(B)q[i] (i=1 to 3) are expressed as q1, q2, and q3. TheCPU 601 determines the rotation matrix R and the translation vector t,used to perform a coordinate transformation so that the targetcoordinate values p1, p2, and p3 completely overlap with the measuredcoordinate values q1, q2, and q3.

The CPU 601 then compares the amount of positional difference determinedin Step S204, with a threshold δ (Step S205). That is, the CPU 601determines whether the amount of positional difference is equal to orsmaller than the threshold. In the first embodiment, the CPU 601determines, as the amount of positional difference, norms |Δp[i]| of thedifference vectors Δp[i] and a norm |t| of the translation vector t. TheCPU 601 then compares the largest value in |Δp[i]| and |t|, with thethreshold δ. Specifically, the CPU 601 determines whether the followingexpression (8) is satisfied.

$\begin{matrix}{{\max \left( {{\max\limits_{i = {({1,2,3})}}{{\Delta \; {p\lbrack i\rbrack}}}},{t}} \right)} \leq \delta} & (8)\end{matrix}$

By using such indexes, the teaching can be performed so that both theamount of positional difference of the marks MK1 to MK3 and the amountof positional difference of the base portion 311 are within a desiredprecision range. Thus, the teaching can be performed also inconsideration of the tilt of the stereo camera 300 with respect to themarks MK1 to MK3. The above-described threshold determination method ismerely one example, and thus any other method may be used to perform thedetermination, by using another index based on the amount of positionaldifference. For example, only the amount of positional difference Δp[i]of positional coordinates of the marks MK1 to MK3 may be used for thethreshold determination, or only the amount of positional differencedetermined by the rotation matrix R and the translation vector t may beused for the threshold determination.

As a result of the determination in Step S205, if the amount ofpositional difference is equal to or smaller than the threshold (S205:Yes), then the CPU 601 causes the display 700 to display “OK” (S206). Ifthe amount of positional difference exceeds the threshold (S205: No),the CPU 601 causes the display 700 to display, for example, the amountand direction of the computed positional difference (S207). In thismanner, the CPU 601 causes the display 700 to indicate information onthe positional differences to a user. The CPU 601 may cause the display700 to display a captured image and superimpose an arrow, whichindicates a required direction and amount of motion of the robot arm201, on the displayed image. In addition, the CPU 601 may cause thedisplay 700 to display, for example, the translation vector t and therotation matrix R computed in Step S204. In this case, not the markpositions but the amount of motion of the palm unit 210 of the robothand 202 is displayed. The amount of motion is a converted amountbetween positions of the palm unit 210. This allows a user tointuitively understand the amount of motion. As another approach forindicating information on the amount of positional difference to a user,blink of a lamp or sound from a speaker may be used as well as thedisplay on the display 700.

After causing the display 700 to display a corresponding display, theCPU 601 determines whether the off-operation is performed by a user(S208). If the off-operation is performed (S208: Yes), then the CPU 601ends the process. If the off-operation is not performed (S208: No), theCPU 601 repeats the steps S201 to S207 until the off-operation isperformed.

While the CPU 601 of the sensor control device 600 performs the processillustrated in FIG. 6, the user checks the display on the display(S107). That is, the user checks whether the display 700 displays “OK”.

If the display 700 does not display “OK” (S107: No), the user operatesthe teaching pendant 800 and adjusts the posture of the robot 200, thatis, the robot arm 201 (S108). That is, the CPU 501 adjusts the postureof the robot 200, that is, the robot arm 201, depending on a commandfrom the teaching pendant 800 operated by the user. While the user movesthe robot 200 by using the teaching pendant 800, the CPU 601 isperforming the process illustrated in FIG. 6. Thus, the CPU 601 updatesthe display on the display 700, in real time.

As a result of the adjustment of the posture of the robot 200, if theamount of positional difference becomes equal to or smaller than thethreshold δ and the display 700 displays “OK” (S107: Yes), the usercauses the HDD 504 to store the posture of the adjusted robot 200, as ateach point (S109). In Step S109, the user may cause the HDD 504 tostore the teach point data, by pressing a button of the teaching pendant800. After teaching the robot 200, the user performs the off-operationby using the input device 850 (S110).

The teach point data corresponds to joint angles of joints of the robotarm 201, that is, the data is angle command values for rotation anglesof motors of the joints. For example, in a case where the robot arm 201has six joints, one piece of teach point data includes six angle commandvalues. The teach point data may include six command values indicating aposition and orientation of the leading end of the robot arm 201, withrespect to the robot coordinate system O. The six command values areobtained by converting six angle command values by using a calculationwhich is based on forward kinematics. In this case, when the robot arm201 is moved in accordance with the teach point, there is required anoperation which converts the teach point data to the angle commandvalues by using a calculation which is based on inverse kinematics.

As described above, in the first embodiment, the calibration is notperformed in a state where the stereo camera 300 is attached to therobot hand 202. Instead, the calibration is performed, in Step S101, inthe state where the stereo camera 300 is detached from the robot hand202. More specifically, the base coordinate system B is defined withrespect to a portion of the positioning pins 321 and 322, which are themechanical reference used when the stereo camera 300 is attached to therobot hand 202; and the calibration is performed by using thecalibration jig 900, with respect to the base coordinate system B. Thus,when the calibration is performed, any error in posture of the robot 200is not included in the calibration result. In general, because jointsand links of the robot arm 201 are distorted depending on its postureand the robot arm 201 has an error produced in its production, there isan error in position and orientation of the hand of the robot 200. Therobot hand 202 also has an error produced in its production. Ifcalibration is performed in a state where the stereo camera 300 isattached to the robot 200, an error in position and orientation of thestereo camera 300 attached to the hand of the robot 200 significantlyaffects the calibration result, reducing accuracy of the relativerelationship R_(c) and t_(c), which serves as calibration data. Incontrast, the first embodiment, which performs the calibration in thestate where the stereo camera 300 is removed from the robot 200, canprevent the error in posture of the robot 200 from affecting therelative relationship R_(c) and t_(c). Thus, the accuracy of therelative relationship R_(c) and t_(c) is increased, and as a result, therobot 200 can be taught with high accuracy.

Since the above-described calibration result is highly accurate, therelative positional relationship (measured values ^(B)q[i]) between themarks MK1 to MK3 and the base coordinate system B can be correctlymeasured. In other words, the position and orientation of the basecoordinate system B with respect to the workpiece holding jig 400 can becorrectly determined. As a result, the amount of positional differenceof the base coordinate system B with respect to the target values^(B)p[i], which is numerically set with respect to the base coordinatesystem B, can be correctly computed in Step S204 without affected by anerror of the robot 200.

Furthermore, the base coordinate system B coincides with the mechanicalreference used when the stereo camera 300 is attached to the robot hand202. Thus, the base coordinate system B can be positioned with respectto the robot hand 202 with high accuracy, by attaching the stereo camera300 to the robot hand 202. As a result, the teaching can be performed inthe state where the robot hand 202 is positioned with respect to theworkpiece holding jig 400 with high accuracy. In this manner, the use ofthe numerically-specified target values ^(B)p[i] enables teaching therobot 200 without any prepared reference image, thus increasingworkability of the teaching.

FIGS. 9A and 9B are explanatory diagrams illustrating a relationshipbetween a tool coordinate system T positioned at a leading end of therobot arm 201 and the base coordinate system B of the stereo camera 300.FIGS. 9A and 9B are different in the position and orientation of therobot hand 202 with respect to the robot arm 201. That is, FIG. 9Aillustrates a configuration in which the robot hand 202 is directlyattached to a flange surface of the robot arm 201, which is the leadingend of the robot arm 201; FIG. 9B illustrates a configuration in whichthe robot hand 202 is attached to the flange surface of the robot arm201 via an attaching jig 204. The tool coordinate system T is acoordinate system positioned at the flange surface of the robot arm 201.

In the first embodiment, the teaching is performed by using the targetvalues ^(B)p[i] of the marks MK1 to MK3 of the workpiece holding jig 400with respect to the base coordinate system B, not by using informationon position and orientation of the robot hand 202 with respect to therobot arm 201. That is, the relationship between the tool coordinatesystem T and the base coordinate system B needs not to be determined.For this reason, the tool coordinate system T needs not to be newly setfor the teaching. As a result, in an event such as changeover ofproduction line, the first embodiment can be easily applied to even acase where the robot hand 202 is attached to the robot arm 201 in adifferent direction in such a manner that the state of FIG. 9A ischanged to the state of FIG. 9B. That is, when the relative position andorientation of the workpiece holding jig 400 seen from the robot hand202, to which the stereo camera 300 is attached, does not change, therobot 200 can be taught again without changing the setting of the targetvalues ^(B)p[i]. Even in such a case, the robot 200 can be taught simplyand with high accuracy.

Second Embodiment

Next, a method of teaching a robot of a robot system according to asecond embodiment will be described. FIG. 10 is an explanatory diagramof the robot system of the second embodiment. A robot system 100Bincludes a robot 200B, a stereo camera 300B which is one example ofvision sensors, the workpiece holding jig 400, the robot control device500, the sensor control device 600, and the display 700. The robotsystem 100B also includes the teaching pendant 800 and the input device850, which are one example of operation devices.

The second embodiment has a configuration different from that of thefirst embodiment in the following points. In the first embodiment, therobot hand 202 of the robot 200 has two fingers; but in the secondembodiment, the robot hand 202B of the robot 200B has three fingers. Inaddition, in the first embodiment, the description has been made for thecase where the robot control device 500 and the sensor control device600 are not connected with each other; but in the second embodiment, therobot control device 500 and the sensor control device 600 areelectrically and communicatively connected with each other via acommunication line. With this configuration, when the teaching isperformed on the robot 200B, that is, on the robot arm 201, the teachingcan be performed not by the jog feed through the operation of theteaching pendant 800 by a user, but by automatic motion of the robot200B.

FIG. 11A is a front view illustrating a state where the stereo camera300B is held by the robot hand 202B in the second embodiment. FIG. 11Bis a side view illustrating the state where the stereo camera 300B isheld by the robot hand 202B in the second embodiment. The robot hand202B is a three-finger robot hand which is suitable to hold acylindrical workpiece for example. The robot hand 202B includes a palmunit 210B which is a hand body, and a plurality of fingers 211B, 212B,and 213B which can open and close with respect to the palm unit 210B. Inthe second embodiment, the number of the fingers is three. With thesecomponents, the robot hand 202B can hold or release a workpiece. Thepalm unit 210B includes a housing and a driving mechanism which ishoused by the housing and configured to drive the fingers 211B, 212B,and 213B.

The stereo camera 300B includes a camera 301B which is a first camera,and a camera 302B which is a second camera. The two cameras, 301B and302B, are disposed in the interior of a housing 310B. The cameras 301Band 302B are digital cameras having an imaging device, such as a CCDimage sensor or a CMOS image sensor. The housing 310B is fixed to aplate-like base portion 311B via an attachment portion 340B. The baseportion 311B is provided with tapered portions 331B, 332B, and 333Bcorresponding to the three-finger robot hand 202B. When the fingers211B, 212B, and 213B are closed in a state where the base portion 311Bis pressed against the palm unit 210B, tapered portions 231B, 232B, and233B of the robot hand 202B engage with the tapered portions 331B, 332B,and 333B of the stereo camera 300B. With this operation, the stereocamera 300B can be chucked with respect to the robot hand 202B. As inthe first embodiment, the palm unit 210B of the robot hand 202B isprovided with the round hole 221 and the elongated hole 222 asillustrated in FIG. 4A. The round hole 221 and the elongated hole 222serve as an engagement portion. As in the first embodiment, the baseportion 311B of the stereo camera 300B is provided with the positioningpins 321 and 322 as illustrated in FIG. 4B. The positioning pins 321 and322 serve as an engagement portion. The positioning pins 321 and 322 andthe holes 221 and 222 are engaged with each other, that is, fit witheach other by the robot hand 202B holding the stereo camera 303B, sothat the stereo camera 300B is positioned with respect to the robot hand202B with high accuracy.

In the second embodiment, even in the case where the robot hand 202B hasthree fingers, the symmetry plane of the stereo optical systems of thestereo camera 300B is the same as a symmetry plane of the positioningmechanisms of the stereo camera 300B which positions the stereo camera300B with respect to the robot hand 202B. The symmetry plane isindicated as a plane D-D. Thus, the symmetry as described in the firstembodiment produces the effect in which the error is prevented fromeasily occurring due to deformation such as distortion of the baseportion 311B.

In the second embodiment, the interface 505 of the robot control device500 and the interface 606 of the sensor control device 600 are connectedwith each other via a communication line. Thus, the CPU 501 and the CPU601 can communicate with each other. In the second embodiment, the twoCPUs, the CPU 501 and the CPU 601, function as a control unit.

FIG. 12 is a flowchart illustrating a method of teaching the robot 200B,according to the second embodiment. Here, the flowchart of FIG. 12includes user operations in addition to the processing performed by theCPUs 501 and 601. In FIG. 12, the CPU 601 first calibrates the relativerelationship between the base coordinate system B and the sensorcoordinate system V (S301: calibration step, calibration process). Thebase coordinate system B has its reference which is at the portion usedto attach the stereo camera 300B to the robot 200B. As in Step S101 ofthe first embodiment, this calibration process is performed in a statewhere the stereo camera 300B is removed from the robot 200B, and set onthe calibration jig 900 illustrated in FIG. 7A. Also in the secondembodiment, the CPU 601 performs the calibration to identify therelative position and orientation between the base coordinate system Band the sensor coordinate system V, and causes the HDD 604 to store therotation matrix R_(c) and the translation vector t_(c), which are thecalibration data. In addition, the CPU 601 sets the target values^(B)p[i] and the threshold δ (S302, S303), as is in the steps S102 andS103 of the first embodiment.

Then, the CPU 501 moves the robot 200B, and causes the robot hand 202Bto hold the stereo camera 300B (S304: attachment step, attachmentprocess). Specifically, a stand (not illustrated) is placed in an areawithin which the robot arm 201 can move, and the stereo camera 300B isplaced on the stand in advance. The CPU 501 causes the robot arm 201 andthe robot hand 202B to move depending on the robot program 510, andcauses the robot hand 202B to automatically hold the stereo camera 300B.With this operation, the stereo camera 300B is positioned with respectto the robot hand 202B with high accuracy. As described above, the CPU501 attaches the stereo camera 300B to the robot 200B by controlling themotion of the robot 200B.

Then, the CPU 501 and the CPU 601 work with each other to refer to therelative relationship R_(c) and t_(c) calibrated and stored in the HDD604 and teach the robot 200B by using the stereo camera 300B (S305 toS313: teaching step, teaching process).

Hereinafter, the description will be made specifically. The CPU 501moves the robot arm 201 to a teach point which has been set in advancethrough an offline teaching (S305). This causes the robot 200B, that is,the robot arm 201 to take a posture in which the teaching is started.That is, in the first embodiment, a user moves the robot 200 by usingthe teaching pendant 800; but in the second embodiment, the CPU 501moves the robot 200B automatically. Then, as in steps S201 to S204described in the first embodiment, the CPU 601 performs capturing images(S306), measuring mark positions (S307), transforming coordinates(S308), and calculating the amount of positional difference (S309). Asin the Step S205 described in the first embodiment, the CPU 601determines whether the amount of positional difference is equal to orsmaller than the threshold (S310).

In the second embodiment, since the robot control device 500 and thesensor control device 600 are electrically connected with each other viaa communication line, the CPU 501 can refer to data on a result of theprocess performed by the CPU 601. If the amount of positional differenceexceeds the threshold (S310: No), the CPU 501 uses calculation data onthe amount of positional difference, the rotation matrix R, and thetranslation vector t, and calculates posture information which is theamount of adjustment for the posture of the robot 200B (S311). The CPU501 then adjusts the posture of the robot 200B depending on the postureinformation (S312). That is, the CPU 501 adjusts the posture of therobot 200B depending on the amount of positional difference. Here,adjusting the posture of the robot 200B means adjusting the angles ofjoints of the robot arm 201, that is, adjusting the position andorientation of the robot hand 202B in the robot coordinate system O.

Hereinafter, the calculation in Step S311 will be specificallydescribed. In the robot control device 500, there is set the toolcoordinate system T whose origin is at the leading end of the robot200B, that is, the robot hand 202B. In addition, there is also presetthe position and orientation of the base coordinate system B withrespect to the tool coordinate system T. Here, the position andorientation of the tool coordinate system T with respect to the robotcoordinate system O is expressed as a matrix ^(O)H_(T). In addition, theposition and orientation of the base coordinate system B with respect tothe tool coordinate system T is expressed as a matrix ^(T)H_(B). Wherethe posture of the adjusted robot 200B is expressed by a matrix^(O)H_(T′), the matrix ^(O)H_(T′) can be expressed by the followingexpression (9).

^(O) H _(T′)=^(O) H _(T)·^(T) H _(B)·^(B) H _(B′)·(^(T) H _(B))⁻¹   (9)

The matrix ^(B)H_(B′) is a homogeneous transformation matrix which iscalculated by using the rotation matrix R and the translation vector tcalculated in Step S203, and which expresses a coordinate transformationfrom a current position to a measured position of the base coordinatesystem B. The matrix ^(B)H_(B′) is expressed by the following expression(10).

$\begin{matrix}{B_{H_{B^{\prime}}} = \begin{bmatrix}R & t \\0 & 1\end{bmatrix}} & (10)\end{matrix}$

That is, in Step S311, the CPU 501 uses the expressions (9) and (10) ,and calculates the matrix ^(O)H_(T′) which expresses the position andorientation of the leading end of the robot 200B. The CPU 501 then usesthe matrix ^(O)H_(T′) and calculates the posture information of therobot 200B, that is, the angle command for joints of the robot arm 201.The CPU 501 then moves the robot 200B depending on the postureinformation, and adjusts the posture of the robot 200B, in S312.

After adjusting the posture of the robot 200B, the CPU 501 performs thesteps S306 to S309 again, and performs the convergence determination inStep S310 by using the threshold δ. If the amount of positionaldifference does not converge in Step S310, the CPU 501 performs thesteps S311, S312, and S306 to S309 again. In this manner, the CPU 501repeats the steps S306 to S312 until the amount of positional differencebecomes equal to or smaller than the threshold δ and converges. That is,the CPU 501 repeatedly causes the robot 200B to perform the motion forposture adjustment of the robot 200B. Since the motion of the robot arm201 and the measurement by the stereo camera 300 have errors, theteaching is often not completed by a single adjustment of posture. Thesecond embodiment can position the robot hand 202B with high accuracy,by repeatedly performing the posture adjustment. As described above, inthe second embodiment, the posture of the robot 200B is automaticallyadjusted by the CPU 501 and the CPU 601 working with each other. Thatis, the CPU 501 and the CPU 601 transform the coordinate values^(V)q[i], measured by using the stereo camera 300B, to the measuredvalues ^(B)q[i] by using the relative relationship R_(c) and t_(c); andadjust the posture of the robot 200B depending on the amount ofpositional difference between the measured values ^(B)q[i] and thetarget values ^(B)p[i].

If the amount of positional difference is equal to or smaller than thethreshold (S310: Yes), the CPU 501 causes the HDD 504 to store theposture of the adjusted robot 200B, that is, the posture of the robotarm 201, as a teach point (S313); and ends the teaching step, that is,the teaching process.

Thus, according to the second embodiment, the automatic motion of therobot arm 201 can cause the robot hand 202B to automatically take therelative position and orientation of the robot hand 202B with respect tothe workpiece holding jig 400.

Also in the second embodiment, the CPU 601 uses visual informationobtained from the stereo camera 300B and the relative relationship R_(c)and t_(c) stored in the HDD 604, and determines the measured values^(B)q[i] of the marks MK1 to MK3 with respect to the base coordinatesystem B. The CPU 501 then adjusts the posture of the robot 200Bdepending on the amount of positional difference between the measuredvalues ^(B)q[i] and the target values ^(B)p[i] with respect to the basecoordinate system B, and performs the teaching. Since the amount ofpositional difference used to adjust the posture of the robot 200B canbe determined without affected by an error in motion of the robot arm201 and an error in production of the robot hand 202B, the robot 200Bcan be taught with high accuracy.

In the second embodiment, Step S311 has been described for the casewhere the matrix ^(T)H_(B), which expresses the relative position andorientation of the base coordinate system B with respect to the toolcoordinate system T of the robot arm 201, is preset. But the presentdisclosure is not limited to this. As described above, because therepetitive operation is performed depending on the adjustment amount,which is based on the amount of positional difference, the matrix^(T)H_(B) to be set may not necessarily be determined with highaccuracy. Even in a case where the matrix ^(T)H_(B) is tilted forexample, by about 30 degrees with respect to a design value, the amountof positional difference converges through the repetitive operation andthe teaching can be performed.

In addition, the matrix ^(T)H_(B) may not be preset. For example, thematrix ^(T)H_(B) can be calculated by automatically performing a knownhand-eye calibration method before the teaching is started. In thiscase, because the robot arm 201 only has to be roughly positioned withrespect to a target position and roughly face the target position, theamount of motion of the robot arm 201 produced when the hand-eyecalibration is performed on the robot arm 201 may be smaller than thecalibration described in the second embodiment. Thus, even when there isa structure around the robot and an area within which the robot can moveis small, the robot can be calibrated and taught.

As another approach, the matrix ^(T)H_(B) may not be used. In this case,the robot arm 201 may be servo-controlled by using a known image-basevisual servo method, so as to reduce the amount of positional differencecomputed by the CPU 601.

Third Embodiment

Next, a method of teaching a robot of a robot system according to athird embodiment will be described. FIG. 13A is an explanatory diagramof a calibration jig 900C of the third embodiment. FIG. 13B is anexplanatory diagram of a robot 200C of the third embodiment. In thefirst and the second embodiments, the description has been made for thecases where the stereo cameras 300 and 300B are, respectively,detachably attached to the robot hands 202 and 202B. In the thirdembodiment, a stereo camera 300C is one example of vision sensors and arobot hand 202C is one example of end effectors. In addition, the stereocamera 300C and the robot hand 202C are fixed to each other as one body.The robot hand 202C, to which the stereo camera 300C is attached as onebody, is detachably attached to a flange surface 201C which is theleading end of the robot 200C. In the third embodiment, the robot 200Cis a vertically articulated robot arm. When the robot hand 202C isattached to the robot 200C, the stereo camera 300C, which is attached tothe robot hand 202C as one body, is attached to the robot 200C. Thus, ajoint portion between the flange surface 201C, which is the leading endof the robot 200C, and the robot hand 202C is a portion which attachesthe stereo camera 300C to the robot 200C.

The calibration jig 900C includes a base portion 901C, pillar portions902C and 903C, a top plate portion 904C, and a reference pattern portion905C. The reference pattern portion 905C is a reference object havingthe same structure as that of the reference pattern portion 905described in the first embodiment. The base portion 901C and the topplate portion 904C are plate-like members which are disposed facing eachother, and joined with each other via the pillar portions 902C and 903C.As illustrated in FIG. 13A, in the third embodiment, the calibration isperformed by positioning the robot hand 202C and the stereo camera 300Cwith respect to the reference pattern portion 905C, and by causing thestereo camera 300C to capture images of the reference pattern portion905C.

In the third embodiment, the origin of the base coordinate system B isset as a position at which the robot hand 202C is attached to the robot200C, that is, a position at which the base coordinate system B has thesame position and orientation as that of the tool coordinate system Twhen the robot hand 202C is attached to the robot 200C. As in the firstembodiment, in the calibration jig 900C, there is measured a positionalrelationship between the base coordinate system B and the marks of thereference pattern portion 905C. With this operation, the relativerelationship which is a relative position and orientation of the sensorcoordinate system V with respect to the base coordinate system B, thatis, the rotation matrix R_(c) and the translation vector t_(c) can bedetermined with high accuracy.

The robot hand 202C is provided with positioning pins 321C and 322Cwhich serve as an engagement portion, and the flange surface 201C of therobot 200C is provided with a round hole 221C and an elongated hole 222Cwhich serve as an engagement portion. Like the round hole 221C and theelongated hole 222C, the base portion 901C is provided with holes 906Cand 907C at positions corresponding to the pins 321C and 322C. With theholes 906C and 907C, the stereo camera 300C can be positioned withrespect to the reference pattern portion 905C with high reproducibility.When the calibration between the sensor coordinate system V and the basecoordinate system B is performed, the stereo camera 300C is placed onthe calibration jig 900C. Thus, the reference pattern portion 905C iscontained in the field of view of the stereo camera 300C. When the robothand 202C is attached to the robot 200C, the pins 321C and 322C engagewith the holes 221C and 222C. Thus, the stereo camera 300C, that is, thebase coordinate system B can be positioned with respect to the robot200C with high accuracy.

The present invention is not limited to the above-described embodiments,and may be variously modified within the technical concept of thepresent invention. In addition, some effects described in theembodiments are merely the most suitable effects produced by the presentinvention. Thus, the effects by the present invention are not limited tothose described in the embodiments.

In the above-described first to third embodiments, the description hasbeen made for the case where the rotation matrix R_(c) and thetranslation vector t_(c), which are calibration data indicating therelative relationship, are stored in a memory such as the HDD 604, andthe teach point data is stored in another memory such as the HDD 504,which is different from the memory such as the HDD 604. That is, thedescription has been made for the case where the calibration data andthe teach point data are stored in different memories. The presentinvention, however, is not limited to this. The calibration data and theteach point data may be stored in a common memory. In addition, thedescription has been made for the case where the calibration data andthe teach point data are stored in the internal memories. The presentdisclosure, however, is not limited to this. The calibration data andthe teach point data may be stored in an external memory. The externalmemory may be a memory directly connected to the robot control device500 or the sensor control device 600, or may be a memory connected tothe robot control device 500 or the sensor control device 600 via anetwork.

In addition, although the first and the second embodiments have beendescribed for the case where the vision sensor is directly attached tothe robot hand of the robot, and the third embodiment has been describedfor the case where the vision sensor is attached to the robot via theend effector which is an intermediate member, the present disclosure isnot limited to this. For example, the vision sensor may be directlyattached to the leading end of the robot arm of the robot.

In addition, although the first and the second embodiments have beendescribed for the case where the engagement portion of the vision sensoris the two positioning pins and the engagement portion of the robot handis the round hole and the elongated hole, the present disclosure is notlimited to this. The engagement portion of the robot and the engagementportion of the vision sensor may have any shape as long as theengagement portions are structured to position the vision sensor withrespect to the mechanical mounting reference of the robot hand. Forexample, one of the robot hand and the vision sensor may be providedwith three abutting surfaces, and the other may be provided withprojection portions which abut against the three abutting surfaces.

In addition, although the first to the third embodiments have beendescribed as examples for the cases where the robot hand has two orthree fingers, the robot may have four or more fingers. In addition,although the end effector is suitably a robot hand, it may not be therobot hand. For example, the end effector may be a welding torch or adriver.

In addition, although the first to the third embodiments have beendescribed for the case where the number of the marks given to themeasured object is three, the present disclosure is not limited to thisnumber. In order to determine the position and orientation of themeasured object, three or more feature points are necessary. Thus, evenin a case where four or more feature points are provided for example,the teaching can also be performed. The increase in the number offeature points smoothes errors when the position and orientation isdetermined by using the least squares method, and thus increasesaccuracy of the teaching.

In addition, although the first to the third embodiments have beendescribed for the case where the vertically articulated robot arm isused, the present disclosure is not limited to this. The presentdisclosure may be applied to a parallel link robot arm, a horizontallyarticulated robot arm, or a Cartesian coordinate robot arm.

In addition, although the first to the third embodiments have beendescribed for the cases where the stereo cameras 300, 300B, and 300C areused as the vision sensor which can measure a three-dimensional positionand orientation, the present disclosure is not limited to this. Forexample, a vision sensor using a method, such as a pattern projectionmethod using a projector, a laser light section method, or aTime-of-Flight method, may be used.

The present invention can also be achieved by providing a program, whichachieves one or more functions of the above-described embodiments, to asystem or a device via a network or a storage medium, and by one or moreprocessors, which are included in computers of the system or the device,reading and executing the program. In addition, the present inventioncan also be achieved by using a circuit, such as an ASIC, which achievesone or more functions.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-022565, filed Feb. 9, 2017, which is hereby incorporated byreference wherein in its entirety.

1-10. (canceled)
 11. A method of controlling a robot system, the methodcomprising: obtaining visual information of an object with a visionsensor; calculating a positional relationship between a first portion ofthe vision sensor and a second portion of a robot by the visualinformation with a controller; and adjusting a teach point of the robotby the positional relationship with the controller.
 12. The methodaccording to claim 11, wherein the controller further calculates thepositional relationship by a calibration data that is obtained beforeobtaining of the visual information.
 13. The method according to claim11, wherein obtaining of the visual information includes capturingimages of an object having three marks.
 14. The method according toclaim 11, wherein the vision sensor includes a plurality of cameras. 15.The method according to claim 11, wherein the vision sensor is a stereocamera.
 16. The method according to claim 11, wherein the vision sensorincludes a camera attached on the robot.
 17. The method according toclaim 11, wherein the controller causes a display to display a result ofthe adjusting.
 18. The method according to claim 11, further comprisingcontrolling the robot by a visual servo method.
 19. A robot systemcomprising a robot and a controller, wherein the controller isconfigured to controls the robot system by the method according to claim1.
 20. The robot system according to claim 19, wherein the robotincludes a force sensor.
 21. A method of controlling a robot system, themethod comprising: obtaining visual information of an object with avision sensor; calculating a positional relationship between a firstportion of the vision sensor and a second portion of a robot by thevisual information with a controller; and adjusting a posture of therobot by the positional relationship with the controller.
 22. The methodaccording to claim 11, further comprising storing the posture of therobot as a teach point.
 23. The method according to claim 21, whereinthe obtaining of the visual information includes capturing images of anobject having at least three marks.
 24. The method according to claim21, wherein the vision sensor includes a plurality of cameras.
 25. Themethod according to claim 21, wherein the vision sensor is a stereocamera.
 26. The method according to claim 21, wherein the vision sensorincludes a camera attached on the robot.
 27. The method according toclaim 21, wherein the controller causes a display to display a result ofthe adjusting.
 28. The method according to claim 21, further comprisingcontrolling the robot by a visual servo method.
 29. A robot systemcomprising a robot and a controller, wherein the controller isconfigured to controls the robot system by the method according to claim21.
 30. The robot system according to claim 29, wherein the robotincludes a force sensor.